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This document, which highlights the efforts of the 
National Institute for Literacy to pilot the development of state 
literacy accountability systems in four states, outlines the 
components, operation, and importance of state accountability systems , 
and discusses considerations in developing a systems approach to 
planning and implementing a quality accountability system. First, the 
institute's mission is explained. Next, four specific areas within 
state literacy systems in which literacy accountability systems build 
capacity are examined along with potential impacts of state literacy 
accountability systems. Discussed next is the role of state literacy 
accountability systems in measuring program results and demonstrating 
that literacy works for selected groups of individuals. Ten steps to 
establishing a state literacy accountability system are listed, and 
demonstration programs in Hawaii, Kentucky, New York, and Tennessee 
are profiled. Each profile includes the following: discussion of 
program's vision, list of program highlights to date, and contact 
person. Lessons learned from the demonstrations are summarized, and 
steps in moving forward from "patchwork quilt" accountability efforts 
to high quality accountability systems are outlined. (MN) 
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"By the year 2000, every adult American will be literate and 
possess the knowledge and skills necessary to compete in a 
global economy and exercise the rights and responsibilities 
of citizenship." 

Coal Six, National Education Coals 

"Literacy is the ability to read something and get the 
information you need and then be able to act. on it." 



Adult Learner 
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Dear Friends and Colleagues, 

Twenty million adult Americans lack the basic literacy skills necessary to be productive 
workers, full partners in their children's education, and citizens involved in building 
strong communities. How can we effectively address this problem? We have to start by 
asking the right questions: 

• How can we best invest in adult literacy? What changes must we make in our 
adult literacy systems to reduce welfare dependence, break the cycle of poverty, 
develop and maintain a quality workforce, assure that children are ready to 
learn, and enable all adults to develop the knowledge and skills necessary to 
participate as active citizens? 

• How can we assure appropriate returns on our investments in literacy? 

These are the key policy and practice questions behind the work of the National 
Institute for Literacy (ND7L). This publication highlights one of our most ambitious 
responses to date. 

Building Capacity by Building System*. The National Institute for Literacy is piloting 
the creation of state literacy accountability systems in four states— Hawai'i, Kentucky, 
New York, and Tennessee. These states are designing and implementing performance 
measurement, reporting, and improvement systems to define the outcomes they seek 
to achieve, monitor results, and make continuous improvements <n literacy-related 
programs and policies. 

Assuring Beturns on Literacy Investment*. Accountability systems enable states to 
unify goals, consolidate programs, develop common terminology and compatible technolo- 
gies, and obtain information essential for more effective decision-making. Even more 
important, effective literacy systems have far-reaching implications for both economic 
development and education reform. 

Documenting that Literacy Work*. At the,Juid-point in this five-year demonstration, 
the achievements are noteworthy and extremely promising for the future. If this 
performance toajectory continues, in the midst of debates about where and how to direct 
scarce public resources, we should be able to document that our literacy investments are 
well-placed, and that, indeed, literacy works. 

Andrew J. Hartman 
Director 

National Institute for Literacy 
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INTRODUCTION 

The National Institute for Literacy (NIFL) is an independent federal agency charged 
with enhancing the national effort to reach the sixth National Education Goal-that all adu t 
Americans will be literate by the year 2000. Created under the National Literacy Act of 1991, 
NIFL has a broad and ambitious mission to: 

♦ Provide national leadership in the development of policies and programs 
that support the creation of a highly effective and efficient literacy system; 

♦ Create an agenda for research, development and evaluation in the area of 
adult and family basic skills education; 

♦ Improve the overall quality of literacy services in the nation to maximize 
progress towards 100% literacy in America; 

♦ Implement an effective communication system that can reach the state, local 
and national levels with timely policy and program information. 

Workinq to Build Capacity and Quality of Services. As the only federal entity adminis- 
tered jointly by the Secretaries of Education, Labor, and Health and Human Services, NIFL is the 
hub of a national network for literacy, working to build capacity and quality of services. One of 
the institute's key capacity building initiatives is the development and piloting of a Performance 
Measurement, Reporting, and Improvement System in four states. What are these state account- 
ability systems, how do they work, and why are they important? 



WHAT IS A STATE ACCOUNTABILITY SYSTEM? 

A Flexible Framework for Systemic Reform. A statewide accountability system is a 
mechanism for focusing on the results of investments in literacy and improving our return on 
those investments. It is a powerful tool for systemic change-a guide for states just beginning 
the change process and a reinforcement for those states already on the way. It builds capacity 
in four specific areas within state literacy systems: 

1 A literacy accountability system focuses efforts to achieve national and state 
goals' A statewide literacy system that aims to enable adults to lead more productive hves 
must be aligned with larger state and federal human resource and economic development 
aoals Developing a statewide accountability system to measure progress toward these goals 
requires interagency cooperation, program collaboration, coordinated service delivery strate- 
gies, and the integrated effort of all stakeholders in the system. 

2 It measures progress by measuring results, not process. Federal and state reporting 
systems have focused in the past on inputs such as the number of clients/students enrolled or 
the number oi hours they attended classes. Such measures tell us little about the real value of 
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the program. A state accountability system guides d state towards denning what it wants to 
achieve— real changes in people's lives. 

3 An accountability system that links literacy to broader state goals puts in place a 
management information system that enables agencies to streamline reporting and 
share information. A state accountability system provides a mechanism for agreement on 
common definitions and elements, and encourages the use of compatible electronic databases 
to centralize information for easy access end updating. Centralized information about pro- 
grams makes possible one-stop program y« pping. Centralized participant files allow students 
to relocate without losing their records. 

4 It ensures the continuous improvement of programs towards 100% results. A state 
literacy accountability system provides information to program managers about how well 
their programs work and to state policymakers about the effectiveness of their policies. Able to 
identify the strengths and weaknesses within their system, states can build on programs and 
strategies that work, discontinue those that don't, and isolate problems that need alternative 
strategies, not additional resources. 



The following chart Illustrates the potential impacts of a state literacy accountability system: 



"WHAT'S IN 



Program Collaboration and Resource 

Coordination 
Measuring Results or Impacts 
Matching Services to Clients' Needs 
Continuous Improvement 
Agreement on Terms 
Providing Accountability 
Centralized Database 
Compatible, State-of-the-Art Technologies 
Programs Linked to State & National Goals 
One-Stop Concept, Multiple Access Points 
Integrated Service Delivery 
Focus on Commonalties 



WHAT'S OUT 



Categorical, Fragmented Programs 

Measuring Inputs 

Matching People to Slots 

Repeating without Rethinking 

Special Terms for each Program 

Reporting Expenditures 

Separate, Uncoordinated Files 

Incompatible, Outdated Technologies 

Programs Satisfy Individual Funding Criteria 

Multiple Sources of Information 

Lack of Coordination Among Service Providers 

Focus on Differences 
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WHY IS A STATE ACCOUNTABILITY 
SYSTEM IMPORTANT? 

For Measuring Results 

A state accountability system helps states know how they are doing in . . . 

♦ Moving toward national and state goals for literacy and lifelong learning 

♦ Building knowledge and skills for adults to compete in a global economy 

♦ Building knowledge and skills to exercise the rights and responsibilities 
of citizenship 

♦ Building knowledge and skills to assure th..t parents are participants in 
their children's education 

To Demonstrate that Literacy Works For. . . 

Adult Learners, who become active participants in their learning process. They gain eas- 
ier access to the system, learn up front about program alternatives, define their own goals, 
and choose the programs to meet those goals. 

♦ As Parents: They set a good example; help children with moral, social, and 
intellectual development; become independent of financial aid, 

♦ As Workers: They get, keep and upgrade jobs; become more flexible workers; teach 
and learn from others in the workplace; and contribute to the productivity of the 
organization, the community, the state, and the nation. 

♦ As Citizens: They become informed citizens; participate in the political process; 
contribute to the community's development; gain citizenship; and vote. 



Service Providers, who know what performance is expected, are able to track 
progress toward goals, and gain greater accountability for results. Streamlined reporting 
requirements mean they spend less time on paperwork and more on client needs. 

State Agencies, who can share resources and information with other state agencies to 
consolidate and streamline existing programs. This information is necessary to evaluate 
and improve programs by identifying barriers and providing incentives for more effective 
policy and program development. 

State and Federal Legislators, who are able to link literacy outcomes to broader state 
and national policies, and gain a better understanding of the value of investment in literacy. 

All Citizens, who see their tax dollars producing efficient and effective literacy programs 
that yield good neighbors, better co-workers and contributing community members. 
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HOW DOES A STATE IMPLEMENT A LITERACY 
ACCOUNTABILITY SYSTEM? 

Unique Approaches. Each state approaches the task differently and adapts the system 
to its unique economic, social, and political conditions. Hawaii, with its strong tradmon of 
SmacrSecisionmaking and consensus building, involved the pubUc and pnvate sectors 
in deling their vision of a literate Hawaii and determining the benchmarks to ^gmde : hem 
here New York, working from the top down and the bottom up, wove together the local per- 
soeSveTnd the policy level perspective to develop their system's outcome statements and 
cooXate wi^ ongomg initiatives in workforce preparation and adult education and tram- 
m7 Kenmcky built its interagency working groups on the solid foundation of a coordinated 
^^Development Cabinet, moving quickly to a pilot effort that wUl base fundmg for- 
mulas and contracts on outcomes. The transformation in Tennessee i ™™ d ^^^ 
work of local task forces, and their insights are, in turn, helping state staff shape pohaes 
that are on track. 



10 STEPS TO ESTABLISH A STATE LITERACY 
ACCOUNTABILITY SYSTEM 



Step 1 Define Vision and Benchmarks 
Step 2 Define Policy Outcomes 

Step 3 Identify Performance Measures for Policy Outcomes 
Step 4 Define Program Outcomes Related to Policy Outcomes 
Step 5 Identify Program Performance Measures 
Step 6 Identify Target Populations Related to Policy and 

Program Outcomes 
Step 7 Compare Existing Service Delivery Patterns With Needs 
Step 8 Collect Data and Communicate Results to Various Literacy 

Stakeholders 

Step 9 Use Information to Improve Program interventions 
Step 10 Use Information to Adjust Resources and Policies 
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Common Values. While each of the four pilot states is pioneering its own approach, a 1 
are building on the same foundation to ensure that their systems are customer-centered qual- 
ity driven, and results-oriented. The chart above presents a synopsis of the key steps each state 
has used to shape and transform its own literacy system. 

Voices of Experience. Following are brief overviews of how the four pilot states are imple- 
menting their own literacy accountability systems. In their own words, they describe what an 
accountability system has enabled them to do. 
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HAWAII 

Vision: In a literate Hawaii, people read, write, 
and communicate at levels that enable them to fulfill 
their aspirations in a secure, prosperous, diverse, and 
healthy society. 

Collaboration From the Start. Developing a 
statewide literacy accountability system has brought 
Hawaii's tradition of collaboration among private and 
non-profit agencies and government to a new level. 
Rather than working together primarily to coordinate 
•programs, the Hawaii Literacy Policy in Action Acad- 
emy and its public and private interagency team have 
worked together from the beginning to create the 
blueprint for a statewide accountability system and a 
vision for a literate Hawaii. Outcomes accountability 
is a powerful tool to shape future policy and programs 
and establish commitments from both the public and private sectors right from the start. 

Greater Awareness of Change. Stakeholders working together to define policy and pro- 
gram outcomes have developed an increased awaneness of change and the change process. 
Translating a new state vision for literacy into actual changes in programs and policies re- 
quires change across the board— from governance to funders' responsibilities to the ability of 
programs to self-monitor and self-correct. 

Becoming a Learning Organization. Becoming a learning organization requires not 
only a willingness to change but a commitment to continuous improvement. An accountabil- 
ity system provides multiple measures which paint a picture of how well things are going and 
which areas require additional attention. The information gathered from multiple measures 
guides both decision-makers and practitioners in the system. 

Highlights to Date: 

♦ Providing training to funders and providers about how to develop and 
respond to Request-for-Proposals that emphasize outcomes accountability. 

♦ Designing planning techniques to enable program staff to self-monitor. 

♦ Setting up process for information sharing across programs while protect- 
ing privacy. 

♦ Analyzing agency data to profile the characteristics of participants, find 
relationships between outcomes and demographics, and discover trends 
that will shape future program design. 

♦ Piloting literacy skills testing and reporting service utilizing the Workplace 
Literacy Test. 

♦ Influencing funders to base funding decisions on outcomes. 

♦ Working closely with all outcome projects in Hawaii to ensure alignment and 
synergy among the projects and ensure that Hawaii's outcomes are realized. 

For more information please contact: Ermile Hargrove, Project Director, Tel: 808*586*7187, 
Fax: 808*586*7191, email: LlT_COUNClL@lib.state.hi.us 
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"NIFL has presented us with 
an excellent opportunity 
for stimulating systemic 
change . . . and creative \ 
thinking. We are becoming ! 
a learning organization. t 
We now have clearly I 
articulated outcomes. " \ 
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KENTUCKY 



Vision: Kentuckians will have the skills, knowl- .^•^-.^ 
edge, and opportunities to compete successfully in the 

state/global economy and to participate fully in fam- "There is no doubt that 

ily and community life. PMRIS will take Kentucky's 

First Steps Began at the Top. Kentucky began by oc / w /f education system 

organizing an Interagency Policy Team, led by the De- miQ tne next centur y y ( 

partment for Adult Education and Literacy, with high ^ ^ ^ ^ 

level membership — commissioner level ana up. The ' 
commitment of the Governor and the State cabinet has more accurately, and as a 

been a key ingredient for success, and has helped team instead of separate 

shape Kentucky's efforts. Kentucky's original perfor- entities across the state 

mance accountability working document was based on fighting battles alone. " 

the Governor's initiatives on healthcare, education, 
and welfare reform. 

Increased Efficiency and Effectiveness. A suc- 
cessful state accountability system establishes perfor- 
mance measures that focus on key outcomes for en- 

hanced quality of life. It also requires the ability to 

collect and transmit data easily and accurately. To accommodate outcomes data collection 
and facilitate program improvement, Kentucky is centralizing and streamlining the data col- 
lection process, moving from bubble sheets to an on-line mainframe system. Centralized, 
technology-based record keeping will save time and effort across the state, and take Kentucky 
into the next century. 

Highlights to Date: 

♦ Determining diverse policy and program outcomes (quality and quantity) 
from family life skills through economic development. 

♦ Determining measures for most outcomes. 

♦ Collecting at least proxy data on majority of outcomes. 

♦ Beginning to build systemic approach with agencies that administer adult 
basic and technical skills. 

♦ Selecting approximately 30 adult education sites for pilot programs. " 

♦ Creating a funding formula based primarily on outcomes. 

♦ Developing a contract for local providers based primarily on outcomes. 

♦ Defining program monitoring and evaluation to assess attainment 
of outcomes. 

♦ Generating active collaboration and participation at state level. 

For more information please contact: Connie Bolin, PMRIS Project Director. 
Telephone: 502»564»4062, Fax: 502»564»6407. 
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NEW YORK 

Vision: New York's population will have the 
knowledge necessary to exercise the rights and re- 
"The most helpful thing for , sponsibilities of citizenship, meet the workforce needs 
New York has been that 0 f a competitive global economy, meet their goals in 

the funding has afforded the workplace, in training programs, in educational 

us the opportunity to settings, and in personal life, and be economically 

explore systems thinking, self sufficient. . v , v , 

H ' r . v Sharing Leadership Across Agencies. New York 

true interagency work, hQs q strQng history Qf coordination across agencies. 

inclusion of local managers Jhe stQte acc0U ntability project has provided the first 

in the development process, opportunity to operate with shared leadership across 

and the opportunity to talk agencies, which has been a positive experience. By 

about the results we are pulling together key agency people at the state and 

trvina to achieve." local levels - there is increased mutual investment. A 

Y y ' non-agency tied project also allows all stakeholders to 

maintain focus and political will. 

Providing the Foundation for an Integrated 
Management System. In New York State there is great interest and belief in the integrated 
human service delivery system, an understanding of what such a system could do for the state, 
and an informal system already operating on the local level, but there is no management sys- 
tem overall. A state accountability system lays the foundation to make possible such a man- 
agement system by putting in place several essential elements— a centralized electronic data 
collection system, a set of terms and definitions common to all human services policies and 
programs, common goals, and a vehicle for planning, measuring and improving. 

Highlights to Date: 

♦ Coordinating component development with other State initiatives 
and projects. 

♦ Devising plan to develop performance-based accountability system for 
adult literacy, basic education, and workforce development system. 

♦ Coordinating and collaborating with the National Governors' Association 
Performance Measurement Project for Workforce Preparation. 

♦ Expanding Policy Group to encompass stakeholders not originally considered. 

♦ Conducting 16 focus interviews at eight sites across the state for input into 
system design. 

♦ Developing working draft of Quantifiable Outcomes at both program and 
policy level through interagency policy level work group and local level in- 
teragency meetings and forums across the state. 

♦ Providing electronic framework to track students and program information. 

For more information please contact: Kate Toms, Project Director/Independent Broker, 
Tel: 518*442*5595, Fax: 518*442*3933. 
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TENNESSEE 



Vision: Twenty-first century Tennessee will have a 
literate citizenry of lifelong learners prospering in 

multi-cultural harmony, respectful of the environment, "We have affirmed that the 

actively participating in the community with healthy best decisions are made by 

fam.hes and economic security for all. people closest to the prob- 

Working Together at State and Local Levels. 7* ' w thrtt t . H 7 
Tennessee's experience is unique in that, from the be- ^ flm ? th °J the / mU5t 

ginning, the TRIMS (Tennessee Reporting and Informa- p,ay ° *J9m™ant role at 

tion System) design has involved both a state-level pol- e ^ ery st . ep ot the wa Y- We 

icy team (representing key agencies, organizations and have 0/50 learne d again 

stakeholders) and four local task forces, bringing to- tnat the support of people 

gether local program managers and staff with other at tn e highest decision- 

key players. Working at both levels simultaneously has making levels in key 

been invaluable in bringing together different perspec- agencies is vital to success. " 

tives on performance management. 

Building on Existing Collaborations. Tennessee 
state agencies involved in adult education and employ- 
ment training already knew each other well and had a track record of collaboration. Develop- 
ing a state accountability system built on and extended existing collaborative efforts. A well- 
developed technology infrastructure greatly enhanced the state's ability to create an integrated 
delivery system. 

Students' Voices Heard Loud and Clear. Transformation of the Tennessee system has 
happened through a local, "bottom up" process. The Student Roundtable of the Tennessee Lit- 
eracy Coalition, along with other student groups at local sites, contributed significantly to 
identifying outcomes from the student perspective. The State Policy Team used these local in- 
sights and the -vork of the local interagency task force members in drafting the State vision 
and policy and program outcomes. The result has been planned outcomes that are realistic 
and tailored to specific populations. 

Highlights To Date 

♦ Creating a working draft of program outcomes and performance measures. 

♦ Involving program participants in describing outcomes of importance to them. 

♦ Creating interagency agreements to share client data at the local level. 

♦ Creating an electronic traveling record to track client progress and facilitate referrals. 

♦ Identifying issues and barriers that prevent effective client service delivery. 

♦ Designing a system that will address barriers. 

♦ Creating interagency working groups at local and state levels. 

For more information please contact: Juliet Merrifield, Project Director, or Mary Ziegler, 
Associate Director, Center for Literacy Studies, The University of Tennessee, Knoxville 
Tel: 615»974»4109, Fax: 615»974»3857. 
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LOOKING BACK ON EXPERIENCE 

Lessons Learned. A performance management, reporting, and improvement system is a 
powerful tool to change paradigms, change systems, revise policies and transform processes- 
all with the goal of wise investment of public funds to change lives for the better. 

1. To achieve national and state goals, it is essential to craft a vision of where you 
want to go, and the goals to get yoa there. A statewide literacy system can be a catalyst to 
bring diverse stakeholders together and enable them to craft a common strategic vision. A 
happy byproduct of this process is buy-in from the contributing stakeholders. 

2. Translating this vision into program change requires broad-based commitment at 
many levels. Moving from the vision of a better life for all to the nuts and bolts of how to do it 
is time consuming and challenging. Shifting the focus from inputs to outcomes doesn't happen 
overnight. Stakeholders at every level need to learn how to think differently about what they 
do. And, once the change in thinking begins, it impacts every aspect of design— denning out- 
comes and determining how to measure them, developing new data collection and manage- 
ment information systems, and improving program perfor- 
mance based on what is learned from the data. 

We want to take down the 3, An effective statewide literacy system must be cus- 

walls between agencies that tomer-focused, results-oriented, and quality-driven. State 

get in the way of providing agencies seeking to build effective statewide literacy systems 

the best services Dossible. s h° uld be 9 in with a focus on the customer-the adult 

x a i T[ ' s^Znnt tn learner-and use their goals and their strengths to help define 

To do that, it is important to ^ outcomes more QCCUrQtely . A client-centered focus 

get all the right players to should also enable agencies to collect and share information 

the table. an( j resources across traditional boundaries. 

— Tennessee 4 Qne hun dr e< i percent results will only be possible if 

other state level human resource and economic development 
initiatives are networked with the literacy system. Literacy 
systems do not exist in a vacuum, but are one piece of the puzzle to help states address multiple 
and overlapping needs. An individual who seeks to develop stronger literacy skills is likely to be 
unemployed or underemployed and in poverty. When we realize that these multiple systems are 
all serving the same client, it is clear that these systems must be networked together. 
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MOVING FORWARD: FROM PATCHWORK QUILT TO 
QUALITY SYSTEMS 

An Unfinished Story. As the four pilot states continue to transform networks for delivery 
of adult education and training from a patchwork of loosely connected, often mismatched 
services to a coordinated, coherent and accessible system, their own processes of continuous 
improvement should yield rich lessons for all of us. As these states gain experience in think- 
ing in new ways, and in defining outcome measures, they will be better able to help others 
make these transformations. As we all analyze progress, and work together to solve complex 
problems, we will improve our ability to help others through the common pitfalls. 

Next Steps. As we see how these statewide literacy systems unfold, we will be better able 
to aggregate and measure the full impact of this initiative. Through these statewide literacy 
accountability systems, we will be better positioned to: 

♦ Share experience and lessons learned with all states; 

♦ Increase understanding of leverage points, incentives, and strategies to 
help states transform their systems; 

♦ Increase the number of states implementing this approach and linking it 
to other statewide efforts in economic development, education reform, 
school-to-work transition, and community collaboration; and 

♦ Ensure continuous improvement and sustainability of these efforts over 
the long term. 

A Foundation for Success. The lessons we have learned thus far have provided a foun- 
dation for success. They have provided some touchstones to let us know we are on the right 
track. And, they have helped all of us see literacy in a new light, as a fundamental building 
block toward achieving national and state education and economic development goals. We 
pledge to continue providing national leadership, building a research agenda, and communi- 
cating our experience to all national, state, and local level stakeholders. 



State leaders cannot afford to rely on guesswork to know if the 
programs they support are in fact achieving intended goals. 
The National Institute for Literacy has invested substantial 
resources in this initiative. Our purpose is to help states build 
adult literacy systems that make a strong and direct impact on 
chronic domestic problems and improve the quality of life for 
their citizens. 
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NIFL FAX-BACK: 202-632-1512 



Please complete this form and fax it back to the National Institute for Literacy. Or if you 
prefer remm S form by mail to the National Institute for Literacy, 800 Connecticut Avenue, 
Suite 200, Washington, DC 20202-7560. 

□ YES! I'm interested in learning more about developing a performance measurement, 
reporting and improvement system. Please send me: 

□ A Guide to Strategic Planning for Performance Measurement, Reporting and 
Improvement Systems 

□ A Review of the Oregon and Texas Experience in Building Performance Measurement 
and Reporting Systems 

j copies of this brochure 

□ YES! I'm interested in learning more about the National Institute for Literacy. Pleaselfend 
me the National Institute for Literacy Newsletter. 

□ Please place me on NIFL's mailing list so that I may keep current on statewide 
literacy initiatives. 



Name . 

Title . 

Organization 
Address 



City/ State /ZIP 

Telephone 

FAX 



If you have questions or would like additional information, 
please call us at 202»632»1500. 
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The National Institute for Literacy 
800 Connecticut Avenue, Suite 200 
Washington, DC 20202-7560 
Tel. (202)»632»1500 Fax (202)»632»1512 
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